2.1 Vector Spaces

Perhaps the most important definition in all of mathematical physics is that of a vector space. The study of vector spaces is called linear algebra, and it is the foundation for almost every area in physics. We will first begin with finite-dimensional vector spaces, and then move on to infinite-dimensional vector spaces in the next chapter.

Table of Contents
Vectors
- Basis and Dimension
Factor Spaces
- Factor Space Basis and Dimension
Direct Sums vs Tensor Products
Summary and Next Steps

Vectors

A vector is an element of a vector space. We denote complex vectors as kets: .

Some preliminary definitions:

A group is a set with a binary operation that satisfies closure, associativity, identity, and invertibility.
An abelian group is a group that also satisfies commutativity.
A field is a set with two binary operations (addition and multiplication) that satisfy the properties of an abelian group under addition, a group under multiplication (excluding zero), and distributivity of multiplication over addition.

Definition 2.1.1 (Vector Space)

A vector space (or linear space) over a field is a non-empty set of objects called vectors, on which two operations are defined: vector addition and scalar multiplication. Members of the field are called scalars.

We often denote a vector space simply as when the field and operations are understood. These operations must satisfy the following axioms for all vectors and scalars:

Vector Addition: For any two vectors , their sum is denoted by . It satisfies the following axioms, making an abelian group:
- Commutativity: for all .
- Associativity: for all .
- Existence of Zero Vector: There exists a a unique vector such that for all .
- Existence of Additive Inverse: For every vector , there exists a unique vector such that .
Scalar Multiplication: For any scalar and vector , their product is denoted by . It satisfies the following axioms:
- Compatibility with Field Multiplication: for all and .
- Identity Element of Scalar Multiplication: for all , where is the multiplicative identity in .
Distributivity:
- Distributivity of Scalar Multiplication with respect to Vector Addition: for all and .
- Distributivity of Scalar Multiplication with respect to Field Addition: for all and .

There are many, many examples of vector spaces. Here are a few.

Example 2.1.2 (Examples of Vector Spaces)

Some examples of vector spaces are:

is a vector space over itself.
is a vector space over itself.
is a vector space over .
is a vector space over .
is not a vector space over , since scalar multiplication is not closed.
The set of oriented line segments in with the same initial point is a vector space over .
The set of all polynomials with real coefficients is a vector space over .
The set of all continuous functions from to is a vector space over .
The set of all matrices with real entries is a vector space over .
The set of all matrices with complex entries is a vector space over .
The set of all solutions to a homogeneous linear differential equation is a vector space over or (This is why you can do things like Fourier analysis and Laplace transforms).

Definition 2.1.3 (Linear Independence)

A set of vectors, , is said to be linearly independent if the only solution to the equation

is the trivial solution .

Definition 2.1.4 (Subspace)

A subset is called a subspace of if is itself a vector space under the same operations of vector addition and scalar multiplication as in .

Example 2.1.5 (Examples of Subspaces)

Some examples of subspaces are:

The set is a subspace of any vector space .
The set itself is a subspace of .
The set of all vectors in that lie on a given plane through the origin is a subspace of .
The set of all polynomials of degree at most is a subspace of the set of all polynomials.
The set of all continuous functions that vanish at a given point is a subspace of the set of all continuous functions.

Theorem 2.1.6 (Span is a Subspace)

Let be a non-empty subset of a vector space . The set of all finite linear combinations of elements of , denoted by , forms a subspace of .

Proof. We need to show that is closed under vector addition and scalar multiplication, and that it contains the zero vector.

Let be a non-empty subset of . By definition, consists of all vectors that can be expressed as finite linear combinations of the vectors in :

Closure under Vector Addition: Let . Then, by definition, there exist scalars such that

and

Adding these two equations, we get

Since are also scalars in , it follows that .
Closure under Scalar Multiplication: Let and . Then, by definition, there exist scalars such that

Now, consider the vector . We have

Since are also scalars in , it follows that .
Existence of Zero Vector: Since is non-empty, let . Then, the zero vector can be expressed as , which is a finite linear combination of elements of . Therefore, .

Since is closed under vector addition and scalar multiplication, and contains the zero vector, it follows that is a subspace of .

Another equivalent definition of the span is the smallest subspace containing . By "smallest", we mean the intersection of all subspaces containing . To prove this, we need to show that the span, as defined by linear combinations, is equal to the intersection of all subspaces containing . We won't do this here, but it is a good exercise. (To prove , show and .)

Subspaces have some nice properties:

The intersection of any collection of subspaces is also a subspace.
The sum of two subspaces and , defined as , is also a subspace.

Basis and Dimension

Definition 2.1.7 (Basis)

A basis of a vector space is a set of vectors that is linearly independent and spans . If the cardinality of the basis is finite, we say that is finite-dimensional. If the cardinality of the basis is infinite, we say that is infinite-dimensional.

Theorem 2.1.8 (Bases Have Equal Cardinality)

For any finite-dimensional vector space , all bases of have the same number of elements.

Proof. Let be a finite-dimensional vector space, and let and be two bases of . We need to show that .

This proof requires some concepts from linear algebra. We will delve into these concepts deeper in later sections, but for now, we will use them without proof. First is the concept of rank and nullity of a matrix.

If we separate a matrix into its column vectors, the rank of the matrix is the dimension of the span of its column vectors. The nullity of the matrix is the dimension of the null space of the matrix, which is the set of all vectors such that . The rank-nullity theorem states that for any matrix with columns, we have

Anways, back to the proof. Since is a basis, it spans . Therefore, each vector can be expressed as a linear combination of the vectors :

for some scalars . This gives us a system of equations in unknowns (the coefficients ).

We can represent this system in matrix form as , where is an matrix whose columns are the vectors , is a vector of coefficients, and is a vector of the vectors . In matrix form it looks like this:

Since is a basis, it is linearly independent. This implies that the matrix has full column rank, meaning that the columns of are linearly independent. Therefore, the rank of is .

By the rank-nullity theorem, we have

where is the dimension of the null space of . Since the columns of are linearly independent, the null space of contains only the zero vector, so . Thus, we have .

Since is an matrix with rank , it follows that . By a symmetric argument, we can also show that . Therefore, we conclude that .

Definition 2.1.9 (Dimension)

If a vector space has a finite basis with elements, we say that is -dimensional, and we write .

Suppose we have a vector , and we want to express it in terms of a basis :

for some scalars . It can be shown that the coefficients are unique, and these are known as the components of the vector in the basis . Importantly, the components depend on the choice of basis! When change of basis is relevant to the discussion, I will sometimes incorporate color coding to help keep track of which basis we are using.

We can also write Equation in matrix form as

where the first matrix is an matrix whose columns are the basis vectors, and the second matrix is an matrix whose entries are the components of .

::example Example 2.1.10 (Examples of Bases)

Some examples of bases are:

In , any nonzero number forms a basis, since any real number can be written as a scalar multiple of . is thus 1-dimensional.
In over (i.e., considering only real scalars), the set forms a basis, since any complex number can be written as a linear combination of and . is thus 2-dimensional over .
In over itself, the set forms a basis, since any complex number can be written as a scalar (complex) multiple of . is thus 1-dimensional over itself.
In 3-dimensional Euclidean space, we have the basis , commonly used in Newtonian physics.
In , the standard basis is given by the vectors , where has a in the -th position and elsewhere.
In the matrix space , the standard basis is given by the matrices , where has a in the position and elsewhere.
In the function space of all infinitely differentiable functions on the interval , the set of monomials forms a basis. By Taylor's theorem, any smooth function can be expressed as a power series in terms of these monomials. Since this basis is infinite, is infinite-dimensional.
Similarly, functions can be expressed in terms of trigonometric functions (Fourier series) or orthogonal polynomials (Legendre, Hermite, etc.), leading to different bases for function spaces.

:::

Factor Spaces

Consider a vector space and a subspace . We can define an equivalence relation on by saying that two vectors are equivalent if their difference is in :

This equivalence relation partitions into equivalence classes. The factor space/quotient set is the set of all these equivalence classes.

Can we turn into a vector space? Yes, we can define vector addition and scalar multiplication on the equivalence classes as follows:

for any scalars and equivalence classes .

Does this definition depend on the choice of representatives and ? To find out, suppose we have another representative and . In order for our definition to be well-defined, we need to show that

But remember that , so it is necessarily equal to for some . Similarly, for some . Therefore, if we subtract from , we get

The right-hand side is within , since is a subspace and is closed under vector addition and scalar multiplication. Thus, our definition of vector addition and scalar multiplication on is well-defined.

Also, since is defined as the set , we sometimes abbreviate it as . With this abbreviation, we have:

Abbreviated Notation	Full Notation

	for all
	for all and
	for all and

Factor Space Basis and Dimension

Let's now find a basis for the factor space . Summarizing what we have so far, we have that is a vector space, is a subspace of , and is the factor space. The factor space is defined as the set of equivalence classes , and we have defined vector addition and scalar multiplication on these equivalence classes. We want to find a basis for .

The idea is to start with a basis for , extend it to a basis for , and then use the additional vectors to form a basis for . Let be a basis for . We can extend this basis to a basis for by adding more vectors such that is a basis for . This means that any vector can be uniquely expressed as

for some scalars . The equivalence class can then be expressed as

But since , we can absorb it into the term, leaving us with

But wait, recall that is just another way of writing the equivalence class . Thus, we can write

Finally, recall that we add and scale equivalence classes by adding and scaling their representatives. Thus, we have

This shows that the set spans . To show that it is linearly independent, suppose we have a linear combination that equals the zero equivalence class:

As the sum is a member of , it can be expressed as a linear combination of the basis vectors :

But wait, the set is a basis for , so it is linearly independent. Thus, the only solution to the above equation is for all and for all . Tracing back, this means that the only solution to is for all . Thus, the set is linearly independent.

Therefore, we have shown that is a basis for .

Also, as a corollary, we have that

This is because if has dimension and has dimension , then the basis for consists of vectors from and additional vectors.

Direct Sums vs Tensor Products

Recall that in set theory, we can define the union and intersection of two sets. In vector spaces, we can define similar operations: the direct sum and the tensor product.

First, the direct sum. Often we like to decompose a vector space into smaller, more manageable pieces. For example, in classical mechanics, we often decompose the motion of a particle into its , , and components. Let the overall vector space be , and let the subspaces be , , and , where each subspace consists of vectors that only have a nonzero component in the respective direction. The "sum" of these subspaces is the set of all vectors that can be written as the sum of vectors from each subspace:

Example 2.1.11 (xy and yz Planes)

Now consider to be the -plane in , and to be the -plane in . We have , since

Definition 2.1.12 (Direct Sum)

If is a vector space, and are subspaces such that and , then we say that is the direct sum of and , denoted by .

A more concrete way to think about the direct sum is this: suppose we write as the set of matrices, and as the set of matrices. Then, we can write the direct sum as the set of matrices, where the first entries come from and the last entries come from .

In other words, the direct sum "stacks" the two subspaces together, but keeps them separate.

Evidently, we have , hence the name "direct sum". We will prove this later.

Proposition 2.1.13 (Characterization of Direct Sum)

If and are subspaces of such that , then if and only if every vector can be uniquely expressed as , where and .

Proof. () In proofs, to show is unique, we define two expressions for and show they are equal.

Suppose . Let and suppose it can be written in two different ways:

for some and . Subtracting the two equations, we get

The left-hand side is in , and the right-hand side is in . The only vector that is in both and is the zero vector, since . Thus, we have and , which implies and . Therefore, the representation of as is unique.

() We need to show that if every vector can be uniquely expressed as , where and , then .

Suppose we have the contradiction that there exists a nonzero vector that is in both and . But then we can write as a sum in infinitely many ways:

Since we can write in infinitely many ways, this contradicts the uniqueness of the representation of vectors in . Therefore, we must have .

Since we have shown both directions, the proposition is proven.

Definition 2.1.14 (Direct Sum of Multiple Subspaces)

If is a vector space, and are subspaces such that and for all , then we say that is the direct sum of , denoted by

Proposition 2.1.15 (Linear Independence in Direct Sum)

Proposition 2.1.15 Let . The set , where for each , is linearly independent.

Intuitively, this proposition makes sense: since the subspaces only intersect at the zero vector, no vector in one subspace can be written as a linear combination of vectors from the other subspaces. For instance, in , the -axis and -axis only intersect at the origin, so no vector on the -axis can be written as a linear combination of vectors on the -axis, and vice versa.

Proof. Suppose we have a set of vectors , where each . Write the direct sum of the subspaces as , which does not necessarily span the entire vector space .

We can always write , where . To show that is linearly independent, consider the equation

for some scalars . If we define , then we can rewrite the equation as

The left-hand side is in , and the right-hand side is in . Since , we must have and . But since , we must have .

We can make the same argument for by cyclically permuting the indices. Thus, we conclude that for all , proving that the set is linearly independent. Extending this to completes the proof.

Proposition 2.1.16 (Existence of Complement Subspace)

If is a subspace of , then there exists a subspace of such that .

Proof. Let be a basis for the subspace .

If , then we can simply take , and we are done. Otherwise, we can extend the basis of to a basis of by adding more vectors such that is a basis for .

Let be the subspace spanned by . They obviously span , since any vector can be written as

for some scalars . Their intersection is only the zero vector, since the basis vectors are linearly independent.

Thus, we have .

Example 2.1.17 (xy Plane Complement)

Consider the vector space and the subspace defined as the -plane, i.e., . A basis for is given by the vectors , where and . To find a subspace such that , we can extend the basis of to a basis of by adding the vector . The subspace is then spanned by , which is the -axis, i.e., . We have , since any vector can be uniquely expressed as

Proposition 2.1.18 (Dimension of Direct Sum)

If , then .

Proof. Let be a basis for , and let be a basis for .

By Proposition 2.1.15, the set is linearly independent. Also, since , this set spans .

Therefore, this set is a basis for .

The number of vectors in this basis is , where and . Thus, we have

One may also find that the direct sum is very similar to the Cartesian product of sets. We can define two vector spaces and , and their Cartesian product as the set of all ordered pairs , where and . We can make into a vector space by defining vector addition and scalar multiplication as follows:

for any , , and .

Proposition 2.1.19 (Isomorphism of Direct Sum and Cartesian Product)

If we associate a vector in as and a vector in as , then we can see that the direct sum is isomorphic to the Cartesian product .

Proof. First, let's show that their intersection is only the zero vector.

Suppose there exists a vector in both and . Then, we can write it as and . Since these two representations must be equal, we have, by the transitivity of equality, that

Thus, the only vector in both and is the zero vector.

Therefore, we have . And as such, we have .

The mapping defined by

is a bijection, as there is one and only one pair for each vector in the direct sum.

Theorem 2.1.20 (Basis of Direct Sum)

Let and be vector spaces with bases and , respectively. Define the set

Then, the set is a basis for the direct sum , which has dimension .

Proof. Before proving it, let's understand what the theorem is saying.

Suppose is -dimensional and can be represented as the set of matrices, and is -dimensional and can be represented as the set of matrices. The new vectors are then

See how the vectors are just the basis vectors of and "stacked" together, with zeros filling in the gaps?

Anyways, let's prove the theorem. First, we need to show that the set is linearly independent.

To do this, we assume a linear combination of the vectors equals the zero vector:

for some scalars . We can rewrite the left-hand side as

Since and are bases for and , respectively, they are linearly independent. Thus, we must have for all and for all .

Therefore, the set is linearly independent.

Next, we need to show that the set spans . To do this, let . Then we can write

for some and . This comes directly from the definition of the direct sum. Since is a basis for , and is a basis for , we can write

for some scalars . But then we can rewrite this as

for some scalars . Thus, the set spans .

Therefore, we have shown that the set is a basis for . Finally, since the basis has vectors, we have .

So, given a Cartesian product , we can always find a corresponding direct sum defined by the addition and scaling laws as in equations and . This direct sum is isomorphic to the Cartesian product, and has dimension equal to the sum of the dimensions of the two vector spaces. There is another set of addition and scaling laws that also define a vector space isomorphic to the Cartesian product, as follows:

The vector space defined over the Cartesian products using these laws is called the tensor product of and , denoted by . For two vectors and , we denote their tensor product as or simply .

If we expand their bases and apply the above laws, we can see that

Thus, the set spans . Moreover, the dimensionality of the tensor product is given by

hence the name "tensor product".

Intuitively, the tensor product can be thought of as a way to combine two vector spaces into a larger one, where the basis vectors of the new space are formed by taking all possible combinations of the basis vectors from the original spaces. In matrix form, it looks like replacing each entry of a matrix with another matrix:

Technically, when we define over vector spaces, that is a tensor product. When we use for elements of vector spaces, that is called the Kronecker product. However, in physics, we often use the same symbol for both operations, and the context usually makes it clear which one is being referred to. We will just use the word "tensor product" to refer to both operations.

Summary and Next Steps

In this section, we have introduced the basics of vector spaces, including their definitions, properties, and examples.

Here are the key points to remember:

A vector space is a set of vectors that can be added together and multiplied by scalars, satisfying certain axioms.
Subspaces are subsets of vector spaces that are also vector spaces.
The span of a set of vectors is the set of all linear combinations of those vectors.
A basis of a vector space is a set of linearly independent vectors that span the entire space.
The dimension of a vector space is the number of vectors in its basis.
The direct sum of subspaces combines them into a larger space while keeping them separate.
The tensor product of vector spaces combines them into a larger space where basis vectors are formed by all possible combinations of the original basis vectors.

Vector spaces alone are not enough to describe physical systems. Much like how we needed metrics to define distances in geometry, we need additional structures to define concepts like length and angle in vector spaces. These additional structures lead us to the concept of inner product spaces, which we will explore in the next section.

Table of Contents​

Vectors​

Basis and Dimension​

Factor Spaces​

Factor Space Basis and Dimension​

Direct Sums vs Tensor Products​

Summary and Next Steps​

Table of Contents